System and method for visualizing placement of a medical tube or line

ABSTRACT

An image processing system is provided. The image processing system includes a display, a processor, and a memory. The memory stores processor-executable code that when executed by the processor causes receiving an image of a region of interest of a patient with an enteric tube or line disposed within the region of interest, detecting the medical tube or line within the image, generating a combined image by superimposing graphical markers on the image that indicate placement or misplacement of the enteric tube or line, and displaying the combined image on a display. In further aspects, a classification of the enteric tube or line (e.g., correctly placed tube present, malpositioned tube present, and so forth) may be determined and communicated to one or more clinicians. Additionally, the outputs of the image processing system may also be provided to facilitate triage of patients, helping prioritize which tube placements require further attention and in what order.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of U.S. ProvisionalApplication No. 63/282,814, entitled “SYSTEM AND METHOD FOR VISUALIZINGPLACEMENT OF A MEDICAL TUBE OR LINE”, filed Nov. 24, 2021, which isherein incorporated by reference in its entirety for all purposes.

TECHNICAL FIELD

The subject matter disclosed herein relates to medical image processing,and more particularly to systems and methods for visualizing placementof a medical tube or line, such as an enteric tube (e.g., a nasogastrictube).

BACKGROUND

Medical imaging may be utilized to visualize medically placed tubes orlines (e.g., chest tube, a nasogastric tube, an enteric tube,endotracheal tube, vascular line, a peripherally inserted centralcatheter (PICC), a catheter, etc.). However, it may be difficult formedical personnel (e.g., doctor, radiologist, technician, etc.) tovisualize these medically placed tubes or lines. In addition, themedical personnel may be untrained or inexperienced, which may hindertheir ability to identify the medically placed tube or line and todetermine if it is properly placed. Further, medical personnel may haveto manually make measurements (which may be time consuming) to determineif a medically placed tube or line is properly placed. If a medicallyplaced tube or line is misplaced, prompt information of suchmisplacement may be desirable in order to take corrective action.

BRIEF DESCRIPTION

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

In certain embodiments, in determining whether a medically placed tubeor line (e.g., an enteric tube or line) is placed properly (e.g., viathe deep learning networks models), a technique is provided includescomparing a measured distance between the surface and/or end of the tubeor line and a reference or anatomical landmark to a desired thresholdand determining if the distance (which may be measured, for example, asthe geometrical distance of two points (e.g., along a straight line) oras a distance measured along the tube curve) is acceptable. The desiredthreshold may represent an acceptable range for the distance between thetube or line and the reference or anatomical landmark for the tube orline to be correctly placed. For example, for a nasogastric tube, thedesired threshold may be a range of distance below the gastroesophagealjunction. If the measured distance is not acceptable, the techniques mayinclude providing a user-perceptible indication of misplacement on adisplay. The indication may be provided on the display where thecombined image is displayed or provided on another device (e.g., theuser's device). The indication may be text stating that the tube or lineis misplaced. In certain embodiments, the text may be more specific andstate the tube or line is too high or too low or otherwise improper. Incertain embodiments, the text may provide further instructions (e.g., toraise or lower the end of the tube or line a certain distance). In someembodiments, the text may be color coded (e.g., in orange or red) tofurther indicate the misplacement. In some embodiments, the indicationmay be provided via color coding of one or more graphical markers or thetube or line displayed on the combined image. For the example, one ormore of the graphical markers (e.g., for the end of tube or line, forthe reference or anatomical landmark, and/or the indication of themeasured distance there between) and/or the tube or line may be colorcoded a specific color (e.g., red or orange) to indicate themisplacement. Alternatively or in addition, one or more of the graphicalmarkers may flash or otherwise be visually highlighted if the tube orline is misplaced. If the measured distance is acceptable, thetechniques may include providing a user-perceptible indication of properplacement of the tube or line. The indication may be provided on thedisplay where the combined image is displayed or provided on anotherdevice (e.g., the user's device). The indication for proper placementmay be text stating the tube or line is properly placed. In certainembodiments, the indication for proper placement may be provided viacolor coding one or more graphical markers of the tube or line displayedon the combined image (e.g., all the graphical markers and/or the tubeor line may be color coded green). In certain embodiments, theindication of proper placement or misplacement may be written into astandard or private information tag (e.g., DICOM) and made visible insubsequent information systems that the image is sent too (e.g., PACS).In certain embodiments, the determination as to whether the medicallyplaced tube or line is properly placed or positioned may be manuallydone by the medical personnel viewing the displayed combined image.

In the context of a nasogastric tube which may be described herein as anexample, with respect to proper placement, a nasogastric tube may beinserted so as to bisect the airways and diaphragm on the X-rayprojection (e.g., to be positioned substantially on the midline withrespect to the airway). The inserted tip (i.e., distal tip) and sideports (if present) are below the diaphragm when properly placed,typically positioned toward the patient's left hand side. Properinsertion and placement of the tube avoids or mitigates possible risks,such as the risk of insertion into the lungs (with the associated riskof substances entering the lungs), the risk of the tube placement beingtoo high, e.g., in the esophagus, and the risk that loops or kinks inthe inserted tube may disturb the flow and/or irritate the patient.

As discussed herein, and in the context of the preceding discussion, thepresently described techniques utilize an AI-based feature to facilitateand assess the placement of enteric tubes, including but not limited tonasogastric tubes. The AI-based feature may be used to detect and/orcharacterize the placed tube, to provide a graphical summary showing thetube with respect to relevant anatomical features (e.g., in the actualanatomical context), and to classify the tube as being placed correctlyor needing adjustment. Use of the AI-based feature may, therefore,increase the confidence of the bedside team when placing tubes. Use ofthe Ai-based feature may also facilitate prioritization of potentiallymisplaced tubes for review, such as by a radiologist, and may speed upthe review process, thereby helping to avoid complications associatedwith misplaced tubes.

Features and benefits provided by the techniques described hereininclude, but are not limited to: the ability to localize particularfeatures (e.g., the tube tip, side port, end port, and so forth) of theenteric tube; the ability to localize relevant anatomical features andcontext (e.g., diaphragm, airways, carina, lungs, patient midline, andso forth); the ability to localize other relevant devices that may bepotentially confounding with enteric tubes (e.g., probes, peripherallyinserted central catheter (PICC) lines, electrocardiogram (ECG) leads orlines, endotracheal (ET) tube, and so forth); the ability to assess thetube position and to provide explanation or commentary about theassessment (e.g., explaining specific problems with current tubeplacement, such as “the side port location is too high relative to thediaphragm”); the ability to assess the tube position and to provideexplanation or commentary regarding aspects of the placement verified tobe correct or satisfactory (e.g., that the tube correctly bisects thediaphragm near the midline); the ability to provide automatedmeasurements that are relevant for the tube assessment (e.g. the lengthof the tube below the diaphragm, the distance of the side port from thediaphragm, the measured tube diameter, etc.); the ability to show thedetected tubes, the tube features, and relevant anatomical features andmeasurements in a graphical summary and the ability to highlightpotentially problematic (or non-problematic) areas within the graphicalsummary); the ability to perform triage based on the tube placementclassification, allowing prioritization of attention to potentiallymisplaced tubes; the ability to save the graphical summary in variousformats (secondary capture, structure report, Integrating the HealthcareEnterprise (IHE) AI Results (AIR), and so forth); and the ability toallow the user to edit, modify, and/or annotate the graphical summary.

In accordance with an embodiment, a medical image processing system isprovided. In accordance with this embodiment, the medical imageprocessing system may comprising: a display; a processor; and a memorystoring processor-executable code. The processor-executable code, whenexecuted by the processor, causes acts to be performed comprising:receiving one or both of a chest or abdominal image of a patient with anenteric tube or line disposed within the region of interest; detectingthe enteric tube or line within the image or images; generating acombined image by superimposing one or more graphical markers on theimage or images wherein the one or more graphical markers indicate oneor more features of enteric tube in an anatomic context; and displayingthe combined image on the display.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the present subjectmatter will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a schematic diagram of an embodiment of a conditioncomparator;

FIG. 2 is a schematic diagram of an embodiment of a clinical analysisapparatus;

FIG. 3 is a schematic diagram of an embodiment of a learning neuralnetwork;

FIG. 4 is a schematic diagram of an embodiment of a particularimplementation of the neural network in FIG. 3 as a convolutional neuralnetwork;

FIG. 5 is a schematic diagram of an embodiment of an image analysisconvolutional neural network;

FIG. 6A is a schematic diagram of an embodiment of a configuration toapply a learning network to process and/or otherwise evaluate an image;

FIG. 6B is a schematic diagram of an embodiment of a combination of aplurality of learning networks;

FIG. 7 is a schematic diagram of an embodiment for training anddeployment phases of a learning network;

FIG. 8 is a schematic diagram of an embodiment of a product leveraging atrained network package to provide a deep learning product offering;

FIGS. 9A-9C are schematic diagrams of respective embodiments of variousdeep learning device configurations;

FIG. 10 is a schematic diagram of an embodiment of an implementation ofthe artificial intelligence classifier of FIG. 2 to process image datato be used by an artificial intelligence model to quantify a condition;

FIG. 11 is a flow diagram of an embodiment of a method for determining aplacement of a medically placed tube or line within a region ofinterest;

FIG. 12 is an example of a process flow of steps taken in the placementand assessment of a nasogastric tube;

FIG. 13 depicts a model architecture for segmenting and classifying anenteric tube placement;

FIG. 14 is an example of a combined image identifying a tube or linewithin a patient;

FIG. 15 is a schematic diagram of a user interface having a combinedimage identifying a tube or line within a patient;

FIGS. 16-20 depict further schematic diagrams of a user interface havinga combined image identifying a tube or line within a patient in acontext where the tube is in an expected position;

FIGS. 21-28 depict further schematic diagrams of a user interface havinga combined image identifying a tube or line within a patient in acontext where the tube is malpositioned;

FIGS. 29-31 depict further schematic diagrams of a user interface havinga combined image identifying a tube or line within a patient in acontext where the tube is in a suboptimal position;

FIGS. 32-34 depict further schematic diagrams of a user interface havinga combined image identifying a tube or line within a patient in acontext where the tube is partially visualized; and

FIG. 35 is a schematic diagram of an embodiment of a processor platformstructured to execute the example machine readable instructions toimplement components disclosed and described herein.

DETAILED DESCRIPTION

One or more specific embodiments will be described below. In an effortto provide a concise description of these embodiments, not all featuresof an actual implementation are described in the specification. Itshould be appreciated that in the development of any such actualimplementation, as in any engineering or design project, numerousimplementation-specific decisions must be made to achieve thedevelopers' specific goals, such as compliance with system-related andbusiness-related constraints, which may vary from one implementation toanother. Moreover, it should be appreciated that such a developmenteffort might be complex and time consuming, but would nevertheless be aroutine undertaking of design, fabrication, and manufacture for those ofordinary skill having the benefit of this disclosure.

When introducing elements of various embodiments of the present subjectmatter, the articles “a,” “an,” “the,” and “said” are intended to meanthat there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements.Furthermore, any numerical examples in the following discussion areintended to be non-limiting, and thus additional numerical values,ranges, and percentages are within the scope of the disclosedembodiments.

Imaging devices (e.g., gamma cameras, positron emission tomography (PET)scanners, computed tomography (CT) scanners, X-Ray machines, fluoroscopymachines, magnetic resonance (MR) imaging machines, ultrasound scanners,etc.) generate medical images (e.g., native Digital Imaging andCommunications in Medicine (DICOM) images) representative of the partsof the body (e.g., organs, tissues, etc.) for various clinical purposes,such as to diagnose and/or treat diseases. Medical images may includevolumetric data including voxels associated with the part of the bodycaptured in the medical image. Medical image visualization softwareallows a clinician to segment, annotate, measure, and/or reportfunctional or anatomical characteristics on various locations of amedical image. In some examples, a clinician may utilize the medicalimage visualization software to identify regions of interest within themedical image.

Acquisition, processing, quality control, analysis, and storage ofmedical image data play an important role in diagnosis and treatment ofpatients in a healthcare environment. A medical imaging workflow anddevices involved in the workflow can be configured, monitored, andupdated throughout operation of the medical imaging workflow anddevices. Machine and/or deep learning can be used to help configure,monitor, and update the medical imaging workflow and devices.

Certain examples discussed herein provide and/or facilitate the use ofimaging devices to provide improved clinical services and outcomes.Certain examples facilitate improved or modified image reconstructionand/or presentation and further processing to provide improved data andanalytics for certain clinical procedures, namely insertion andplacement of a medical line or tube, such as an enteric tube (e.g., anasogastric tube).

Certain examples provide an image processing apparatus including anartificial intelligence classifier. The classifier can detect, segment,and quantify anatomic features and/or medical devices, for example. Theclassifier can be a discrete output of positive or negative for afinding, a segmentation, etc. For example, the classifier caninstantiate machine learning and/or other artificial intelligence todetect, segment, and analyze a presence of a medical device (e.g.,medically placed tube or line). By way of example, the classifier caninstantiate machine learning and/or other artificial intelligence todetect an end of a medically placed tube or line (such as an enterictube), detect a reference or anatomical landmark, determine a positionof the medically placed tube or line relative to the reference oranatomical landmark, measure a distance between the end of the medicallyplaced tube or line and the reference landmark, and determine whetherthe tube or line is properly placed.

Machine learning techniques, whether deep learning networks or otherexperiential/observational learning system, can be used to locate anobject in an image, understand speech and convert speech into text, andimprove the relevance of search engine results, for example. Deeplearning is a subset of machine learning that uses a set of algorithmsto model high-level abstractions in data using a deep graph withmultiple processing layers including linear and non-lineartransformations. While many machine learning systems are seeded withinitial features and/or network weights to be modified through learningand updating of the machine learning network, a deep learning networktrains itself to identify “good” features for analysis. Using amultilayered architecture, machines employing deep learning techniquescan process raw data better than machines using conventional machinelearning techniques. Examining data for groups of highly correlatedvalues or distinctive themes is facilitated using different layers ofevaluation or abstraction.

Throughout the specification and claims, the following terms take themeanings explicitly set forth herein, unless the context dictatesotherwise. The term “deep learning” is a machine learning technique thatutilizes multiple data processing layers to recognize various structuresin data sets and classify the data sets with high accuracy. A deeplearning network can be a training network (e.g., a training networkmodel or device) that learns patterns based on a plurality of inputs andoutputs. A deep learning network can be a deployed network (e.g., adeployed network model or device) that is generated from the trainingnetwork and provides an output in response to an input.

The term “supervised learning” is a deep learning training method inwhich the machine is provided already classified data from humansources. The term “unsupervised learning” is a deep learning trainingmethod in which the machine is given data that has not been previouslyclassified for training. Such unsupervised learning techniques may besuitable for training that is directed to abnormality detection. Theterm “semi-supervised learning” is a deep learning training method inwhich the machine is provided a small amount of classified data fromhuman sources compared to a larger amount of unclassified data availableto the machine.

The term “representation learning” is a field of methods fortransforming raw data into a representation or feature that can beexploited in machine learning tasks. In supervised learning, featuresare learned via labeled input.

The term “convolutional neural networks” or “CNNs” are biologicallyinspired networks of interconnected data used in deep learning fordetection, segmentation, and recognition of pertinent objects andregions in datasets. CNNs evaluate raw data in the form of multiplearrays, breaking the data into a series of stages and examining the datafor learned features.

The term “transfer learning” is a process of a machine storing theinformation used in properly or improperly solving one problem to solveanother problem of the same or similar nature as the first. Transferlearning may also be known as “inductive learning”. Transfer learningcan make use of data from previous tasks, for example.

The term “active learning” is a process of machine learning in which themachine selects a set of examples for which to receive training data,rather than passively receiving examples chosen by an external entity.For example, as a machine learns, the machine can be allowed to selectexamples that the machine determines will be most helpful for learning,rather than relying only an external human expert or external system toidentify and provide examples.

The term “computer aided detection” or “computer aided diagnosis” referto computers that analyze medical images for the purpose of detecting ananatomic structure of interest, a physiological measurement or event ofinterest, and/or of suggesting a possible diagnosis.

Certain examples use neural networks and/or other machine learningarchitectures to implement a new workflow for image and associatedpatient analysis including generating alerts based on radiologicalfindings that may be generated and delivered at the point of care of aradiology exam. Certain examples use Artificial Intelligence (AI)algorithms to process one or more imaging exams (e.g., an image or setof images), and provide an alert based on the automated exam analysis.The alert(s) (e.g., including notification(s), recommendation(s), otheraction(s), etc.) may be intended for the technologist performing theexam, clinical team providers (e.g., nurse, doctor, etc.), radiologist,administration, operations, and/or even the patient. The alerts may beprovided to indicate a specific, or multiple, quality control issueand/or radiological finding(s) or lack thereof in the exam image data,for example.

In certain examples, the AI algorithm can be (1) embedded within animaging device, (2) running on a mobile device (e.g., a tablet, smartphone, laptop, other handheld or mobile computing device, etc.), and/or(3) running in a cloud computing architecture (e.g., on premise or offpremise) and delivers the alert via a web browser (e.g., which mayappear on the radiology system, mobile device, computer, etc.). Suchconfigurations can be vendor neutral and compatible with legacy imagingsystems. For example, if the AI processor is running on a mobile deviceand/or in the “cloud”, the configuration can receive the images (A) fromthe x-ray and/or other imaging system directly (e.g., set up assecondary push destination such as a Digital Imaging and Communicationsin Medicine (DICOM) node, etc.), (B) by tapping into a Picture Archivingand Communication System (PACS) destination for redundant image access,(C) by retrieving image data via a sniffer methodology (e.g., to pull aDICOM image off the system once it is generated), etc.

Certain examples provide apparatus, systems, methods, etc., to determineand provide, as discussed herein, clinical feedback relevant to thetreatment or care of a patient, such as placement of a clinical line ortube, and/or other patient-relevant conditions based on output of analgorithm instantiated using and/or driven by an artificial intelligence(AI) model, such as a deep learning network model, machine learningnetwork model, etc. For example, the presence of a medically placed tubeor line (e.g. chest tube, an enteric tube (such as a nasogastric tube),endotracheal tube, vascular line, a peripherally inserted centralcatheter, a catheter, etc.) can be determined based on an output of anAI detection algorithm. In addition, the placement of a medical tube orline within a region of interest (e.g., lung, stomach, vascular system,etc.) can be determined based on an output of an AI detection (e.g.,whether the medical tube or line is properly placed).

Certain examples discussed and described herein provide systems andmethod to detect a medically placed tube or line within a region ofinterest of a patient and whether the tube or line is properly placedwithin the region of interest based on an AI classification algorithmapplied to a patient's data. An example method includes detecting apresence of a medically placed tube or line in an image; detecting anone or more a terminal end and/or surface contours of the medicallyplaced tube or line in the image; detecting a reference one or moreanatomical landmarks in the image; determining whether the medicallyplaced tube or line is properly placed relative to the reference oranatomical landmark(s); and/or providing a notification for a caregiveras to whether the medically placed tube or line is properly placedrelative to the reference or anatomical landmark(s). In certainembodiments, the AI classification algorithm may detect the presence ofthe medically placed line or tube; graphically mark the medically placedline or tube with a visual (e.g., color or color-coded) graphicaloverlay; detect a surface and/or end (e.g., distal end) of the medicallyplaced line or tube; graphically mark the surface and/or end of themedically placed tube or line; detect one or more reference oranatomical landmarks (e.g., for determining the proper placement of thetube or line relative to the landmark(s)); graphically mark thereference or anatomical landmark(s); calculate a distance between thesurface and/or end of the medically placed tube or line; and/orcalculate and provide a confidence metric or other metric (e.g., for thecalculated distance, for the determination of the presence of themedically placed tube or line, for an accuracy in detecting the end ofthe tube or line, for an accuracy in detecting the reference oranatomical landmark, etc.). The AI classification algorithm is trainedbased on images with or without medically placed tubes or lines, imageswith properly placed tubes or lines, images with misplaced tubes orlines, images with the reference or anatomical landmark, and/or imageswithout the reference or anatomical landmark.

For example, patients in a critical care setting receive x-rays (e.g.,chest sx-rays) to monitor the placement of a medically placed tube orline. If a tube or line is misplaced, the medical team may need toconduct an intervention to properly place the medical tube or line. Anartificial intelligence classifier can detect a presence of themedically placed tube or line, detect the surface and/or terminal end ofthe medically placed tube or line, detect a reference or anatomicallandmark, and evaluate whether the tube or line is properly placed. Analert can be generated and output at a point of care, such as on adevice (e.g., an imaging device, an imaging workstation, etc.), tonotify and/or otherwise provide instructions (e.g., notification that atube is or is not properly placed or instruction to remove the tube orline, shift the tube or line in a certain direction, etc.) to a clinicalcare team, for example.

The techniques describe herein provide a quick means to determine if amedically placed tube or line is improperly placed. This enables afaster intervention to ensure the tube or line is in an appropriatelocation for patient care. In addition, it relieves some of the burdenon the medical team providing assistance to the patient (especiallythose personnel who may be untrained or inexperienced).

Deep learning is a class of machine learning techniques employingrepresentation learning methods that allows a machine to be given rawdata and determine the representations needed for data classification.Deep learning ascertains structure in data sets using backpropagationalgorithms which are used to alter internal parameters (e.g., nodeweights) of the deep learning machine. Deep learning machines canutilize a variety of multilayer architectures and algorithms. Whilemachine learning, for example, involves an identification of features tobe used in training the network, deep learning processes raw data toidentify features of interest without the external identification.

Deep learning in a neural network environment includes numerousinterconnected nodes referred to as neurons. Input neurons, activatedfrom an outside source, activate other neurons based on connections tothose other neurons which are governed by the machine parameters. Aneural network behaves in a certain manner based on its own parameters.Learning refines the machine parameters, and, by extension, theconnections between neurons in the network, such that the neural networkbehaves in a desired manner.

Deep learning that utilizes a convolutional neural network segments datausing convolutional filters to locate and identify learned, observablefeatures in the data. Each filter or layer of the CNN architecturetransforms the input data to increase the selectivity and invariance ofthe data. This abstraction of the data allows the machine to focus onthe features in the data it is attempting to classify and ignoreirrelevant background information.

Deep learning operates on the understanding that many datasets includehigh level features which themselves include low level features. Whileexamining an image, for example, rather than looking for an object, itis more efficient to look for edges which form motifs which form parts,which form the object being sought. These hierarchies of features can befound in many different forms of data such as speech and text, etc.

Learned observable features include objects and quantifiableregularities learned by the machine during supervised learning. Amachine provided with a large set of well classified data is betterequipped to distinguish and extract the features pertinent to successfulclassification of new data.

A deep learning machine that utilizes transfer learning may properlyconnect data features to certain classifications affirmed by a humanexpert. Conversely, the same machine can, when informed of an incorrectclassification by a human expert, update the parameters forclassification. Settings and/or other configuration information, forexample, can be guided by learned use of settings and/or otherconfiguration information, and, as a system is used more (e.g.,repeatedly and/or by multiple users), a number of variations and/orother possibilities for settings and/or other configuration informationcan be reduced for a given situation.

An example deep learning neural network can be trained on a set ofexpert classified data, classified and further annotated for objectlocalization, for example. This set of data builds the first parametersfor the neural network, and this would be the stage of supervisedlearning. During the stage of supervised learning, the neural networkcan be tested whether the desired behavior has been achieved.

Once a desired neural network behavior has been achieved (e.g., amachine has been trained to operate according to a specified threshold,etc.), the machine can be deployed for use (e.g., testing the machinewith “real” data, etc.). During operation, neural networkclassifications can be confirmed or denied (e.g., by an expert user,expert system, reference database, etc.) to continue to improve neuralnetwork behavior. The example neural network is then in a state oftransfer learning, as parameters for classification that determineneural network behavior are updated based on ongoing interactions. Incertain examples, the neural network can provide direct feedback toanother process. In certain examples, the neural network outputs datathat is buffered (e.g., via the cloud, etc.) and validated before it isprovided to another process.

Deep learning machines using convolutional neural networks (CNNs) can beused for image analysis. Stages of CNN analysis can be used for facialrecognition in natural images, computer-aided diagnosis (CAD), etc.

High quality medical image data can be acquired using one or moreimaging modalities, such as x-ray, computed tomography (CT), molecularimaging and computed tomography (MICT), magnetic resonance imaging(MRI), etc. Medical images are largely interpreted by physicians, butthese interpretations can be subjective, affected by the condition ofthe physician's experience in the field and/or fatigue. Image analysisvia machine learning can support a healthcare practitioner's workflow.

Deep learning machines can provide computer aided detection support toimprove their image analysis with respect to image quality andclassification, for example. However, issues facing deep learningmachines applied to the medical field often lead to numerous falseclassifications. Deep learning machines must overcome small trainingdatasets and require repetitive adjustments, for example.

Deep learning machines, with minimal training, can be used to determinethe quality of a medical image, for example. Semi-supervised andunsupervised deep learning machines can be used to quantitativelymeasure qualitative aspects of images. For example, deep learningmachines can be utilized after an image has been acquired to determineif the quality of the image is sufficient for diagnosis. Supervised deeplearning machines can also be used for computer aided diagnosis.Supervised learning can help reduce susceptibility to falseclassification, for example.

Deep learning machines can utilize transfer learning when interactingwith physicians to counteract the small dataset available in thesupervised training. These deep learning machines can improve theircomputer aided diagnosis over time through training and transferlearning.

FIG. 1 illustrates an example condition comparator apparatus 100including a plurality of inputs 110, 115, an artificial intelligence(AI) classifier 120, and an output comparator 130. Each input 110, 115is provided to the AI classifier 120, which classifies image data and/orother information in the respective input 110, 115 to identify acondition in the input 110, 115 and to generate an indication of theidentified condition based on the input 110, 115. In certainembodiments, the AI classifier 120 may classify images and/or otherinformation in the respective input 110, 115 to identify a medicallyplaced tube or line (e.g., chest tube, enteric tube, endotracheal tube,vascular line, a peripherally inserted central catheter, a catheter,etc.) and to identify a reference or anatomical landmark relevant to thetype or line and its desired placement. Using the example comparatorapparatus 100, it can be determined whether the tube or line is properlyplaced within a region of interest of the patient relative to areference or anatomical landmark. In particular, both surfaces and/or anend of the tube or line as well as a reference or anatomical landmarkmay be located and a determination made as to whether the tube or lineis properly placed relative to the reference or anatomical landmark. Adistance may be measured between the end of the tube or line and thereference or anatomical landmark in determining whether the end of thetube or line is properly placed. A confidence metric (e.g., for thecalculated distance, for the determination of the presence of themedically placed tube or line, for an accuracy in detecting the end ofthe tube or line, for an accuracy in detecting the reference oranatomical landmark, etc.) may be calculated and/or provided viauser-perceptible notification or stored for further reference. Further,a notification or alert may be provided as to whether or not themedically placed tube or line is properly placed. If the tube or line isnot properly placed, further instructions may be provided via anotification or alert (e.g., related to moving the tube or line in acertain direction).

FIG. 2 illustrates an example clinical progression analysis apparatus200 that can be constructed based on the example condition comparator100 of FIG. 1 . The example apparatus 200 includes a data source 210, anartificial intelligence (AI) classifier 220, a data store 230, acomparator 240, an output generator 250, and a trigger 260. Input 110,115 can be provided by the data source 210 (e.g., a storage device, animaging device, etc., incorporated in and/or otherwise connected to theapparatus 200, etc.) to the AI classifier 220.

The example classifier 220 processes input over time to correlate inputfrom the data source 210 with a classification. Thus, the AI classifier220 processes input image data and/or other data to identify a conditionin the input data and classify that condition according to one or morestates (e.g., tube or line present, tube or line not present, referenceor anatomical landmark present, reference or anatomical landmark notpresent, tube or line placed correctly, tube or line misplaced) asspecified by an equation, a threshold, and/or other criterion. Incertain embodiments, the AI classifier 220 processes input image dataand/or other data to detect a medically placed tube or line and todetermine whether an end of the medically placed tube or line isproperly placed. Output of the AI classifier 220 can be stored in thedata store 230, for example.

Over time, classifications made by the AI classifier 220 with respect tothe same type of input 110, 115 from the data source 210 (e.g., lung MRimages of the same patient taken at times t0 and t1, etc.) can begenerated and stored in the data store 230. The classifications areprovided to the comparator 240, which compares a classification at twoor more different times (e.g., prior to insertion of the tube or lineand after the insertion of the tube or line) to identify the medicallyplaced tube or line and determine whether the medically placed tube orline is properly placed. For example, at time t0 the tube or line maynot present in the region of interest and at time t1 or a later time thetube or line may be placed in a location (which may or may not beproperly placed) within the region of interest.

The comparator 240 provides a result indicative of thetrend/progression. In certain embodiments, the comparator 240 provides aresult indicative of a placement of a medically placed tube or line. Theoutput generator 250 transforms that result into an output that can bedisplayed, stored, provided to another system for further processingsuch as an alert, a notification or order, an adjustment in patientcare, (e.g., a point of care alert system, an imaging/radiologyworkstation, a computer-aided diagnosis (CAD) processor, a schedulingsystem, a medical device, etc.), etc.

The trigger 260 coordinates actions among the data source 210, the AIclassifier 220, the data store 230, the comparator 240, and the outputgenerator 250. The trigger 260 can initiate input of data from the datasource 210 to the classifier 220, comparison of results from the datastore 230 by the comparator 240, output by the output generator 250.Thus, the trigger 260 serves as a coordinator among elements of theapparatus 200.

FIG. 3 is a representation of an example learning neural network 300.The example neural network 300 includes layers 320, 340, 360, and 380.The layers 320 and 340 are connected with neural connections 330. Thelayers 340 and 360 are connected with neural connections 350. The layers360 and 380 are connected with neural connections 370. Data flowsforward via inputs 312, 314, 316 from the input layer 320 to the outputlayer 380 and to an output 390.

The layer 320 is an input layer that, in the example of FIG. 3 ,includes a plurality of nodes 322, 324, 326. The layers 340 and 360 arehidden layers and include, in the example of FIG. 3 , nodes 342, 344,346, 348, 362, 364, 366, 368. The neural network 300 may include more orless hidden layers 340 and 360 than shown. The layer 380 is an outputlayer and includes, in the example of FIG. 3 , a node 382 with an output390. Each input 312-316 corresponds to a node 322-326 of the input layer320, and each node 322-326 of the input layer 320 has a connection 330to each node 342-348 of the hidden layer 340. Each node 342-348 of thehidden layer 340 has a connection 350 to each node 362-368 of the hiddenlayer 360. Each node 362-368 of the hidden layer 360 has a connection370 to the output layer 380. The output layer 380 has an output 390 toprovide an output from the example neural network 300.

Of connections 330, 350, and 370 certain example connections 332, 352,372 may be given added weight while other example connections 334, 354,374 may be given less weight in the neural network 300. Input nodes322-326 are activated through receipt of input data via inputs 312-316,for example. Nodes 342-348 and 362-368 of hidden layers 340 and 360 areactivated through the forward flow of data through the network 300 viathe connections 330 and 350, respectively. Node 382 of the output layer380 is activated after data processed in hidden layers 340 and 360 issent via connections 370. When the output node 382 of the output layer380 is activated, the node 382 outputs an appropriate value based onprocessing accomplished in hidden layers 340 and 360 of the neuralnetwork 300.

FIG. 4 illustrates a particular implementation of the example neuralnetwork 300 as a convolutional neural network 400. As shown in theexample of FIG. 4 , an input 310 is provided to the first layer 320which processes and propagates the input 310 to the second layer 340.The input 310 is further processed in the second layer 340 andpropagated to the third layer 360. The third layer 360 categorizes datato be provided to the output layer 380. More specifically, as shown inthe example of FIG. 4 , a convolution 404 (e.g., a 5×5 convolution,etc.) is applied to a portion or window (also referred to as a“receptive field”) 402 of the input 310 (e.g., a 32×32 data input, etc.)in the first layer 320 to provide a feature map 406 (e.g., a (6×) 28×28feature map, etc.). The convolution 404 maps the elements from the input310 to the feature map 406. The first layer 320 also providessubsampling (e.g., 2×2 subsampling, etc.) to generate a reduced featuremap 410 (e.g., a (6×) 14×14 feature map, etc.). The feature map 410undergoes a convolution 412 and is propagated from the first layer 320to the second layer 340, where the feature map 410 becomes an expandedfeature map 414 (e.g., a (16×) 10×10 feature map, etc.). Aftersubsampling 416 in the second layer 340, the feature map 414 becomes areduced feature map 418 (e.g., a (16×) 4×5 feature map, etc.). Thefeature map 418 undergoes a convolution 420 and is propagated to thethird layer 360, where the feature map 418 becomes a classificationlayer 422 forming an output layer of N categories 424 with connection426 to the convoluted layer 422, for example.

FIG. 5 is a representation of an example implementation of an imageanalysis convolutional neural network 500. The convolutional neuralnetwork 500 receives an input image 502 and abstracts the image in aconvolution layer 504 to identify learned features 510-522. In a secondconvolution layer 530, the image is transformed into a plurality ofimages 530-538 in which the learned features 510-522 are eachaccentuated in a respective sub-image 530-538. The images 530-538 arefurther processed to focus on the features of interest 510-522 in images540-548. The resulting images 540-548 are then processed through apooling layer which reduces the size of the images 540-548 to isolateportions 550-554 of the images 540-548 including the features ofinterest 510-522. Outputs 550-554 of the convolutional neural network500 receive values from the last non-output layer and classify the imagebased on the data received from the last non-output layer. In certainexamples, the convolutional neural network 500 may contain manydifferent variations of convolution layers, pooling layers, learnedfeatures, and outputs, etc.

FIG. 6A illustrates an example configuration 600 to apply a learning(e.g., machine learning, deep learning, etc.) network to process and/orotherwise evaluate an image. Machine learning can be applied to avariety of processes including image acquisition, image reconstruction,image analysis/diagnosis, etc. As shown in the example configuration 600of FIG. 6A, raw data 610 (e.g., raw data 610 such as sonogram raw data,etc., obtained from an imaging scanner such as an x-ray, computedtomography, ultrasound, magnetic resonance, etc., scanner) is fed into alearning network 620. The learning network 620 processes the data 610 tocorrelate and/or otherwise combine the raw data 620 into processed data630 (e.g., a resulting image, etc.) (e.g., a “good quality” image and/orother image providing sufficient quality for diagnosis, etc.). Thelearning network 620 includes nodes and connections (e.g., pathways) toassociate raw data 610 with the processed data 630. The learning network620 can be a training network that learns the connections and processesfeedback to establish connections and identify patterns, for example.The learning network 620 can be a deployed network that is generatedfrom a training network and leverages the connections and patternsestablished in the training network to take the input raw data 610 andgenerate the resulting image 630, for example.

Once the learning 620 is trained and produces good images 630 from theraw image data 610, the network 620 can continue the “self-learning”process and refine its performance as it operates. For example, there is“redundancy” in the input data (raw data) 610 and redundancy in thenetwork 620, and the redundancy can be exploited.

If weights assigned to nodes in the learning network 620 are examined,there are likely many connections and nodes with very low weights. Thelow weights indicate that these connections and nodes contribute littleto the overall performance of the learning network 620. Thus, theseconnections and nodes are redundant. Such redundancy can be evaluated toreduce redundancy in the inputs (raw data) 610. Reducing input 610redundancy can result in savings in scanner hardware, reduced demands oncomponents, and also reduced exposure dose to the patient, for example.

In deployment, the configuration 600 forms a package including an inputdefinition 610, a trained network 620, and an output definition 630. Thepackage can be deployed and installed with respect to another system,such as an imaging system, analysis engine, etc. An image enhancer 625can leverage and/or otherwise work with the learning network 620 toprocess the raw data 610 and provide a result (e.g., processed imagedata and/or other processed data 630, etc.). The pathways andconnections between nodes of the trained learning network 620 enable theimage enhancer 625 to process the raw data 610 to form the image and/orother processed data result 630, for example.

As shown in the example of FIG. 6B, the learning network 620 can bechained and/or otherwise combined with a plurality of learning networks621-623 to form a larger learning network. The combination of networks620-623 can be used to further refine responses to inputs and/orallocate networks 620-623 to various aspects of a system, for example.

In some examples, in operation, “weak” connections and nodes caninitially be set to zero. The learning network 620 then processes itsnodes in a retaining process. In certain examples, the nodes andconnections that were set to zero are not allowed to change during theretraining. Given the redundancy present in the network 620, it ishighly likely that equally good images will be generated. As illustratedin FIG. 6B, after retraining, the learning network 620 becomes DLN 621.The learning network 621 is also examined to identify weak connectionsand nodes and set them to zero. This further retrained network islearning network 622. The example learning network 622 includes the“zeros” in learning network 621 and the new set of nodes andconnections. The learning network 622 continues to repeat the processinguntil a good image quality is reached at a learning network 623, whichis referred to as a “minimum viable net (MVN)”. The learning network 623is a MVN because if additional connections or nodes are attempted to beset to zero in learning network 623, image quality can suffer.

Once the MVN has been obtained with the learning network 623, “zero”regions (e.g., dark irregular regions in a graph) are mapped to theinput 610. Each dark zone is likely to map to one or a set of parametersin the input space. For example, one of the zero regions may be linkedto the number of views and number of channels in the raw data. Sinceredundancy in the network 623 corresponding to these parameters can bereduced, there is a highly likelihood that the input data can be reducedand generate equally good output. To reduce input data, new sets of rawdata that correspond to the reduced parameters are obtained and runthrough the learning network 621. The network 620-623 may or may not besimplified, but one or more of the learning networks 620-623 isprocessed until a “minimum viable input (MVI)” of raw data input 610 isreached. At the MVI, a further reduction in the input raw data 610 mayresult in reduced image 630 quality. The MVI can result in reducedcomplexity in data acquisition, less demand on system components,reduced stress on patients (e.g., less breath-hold or contrast), and/orreduced dose to patients, for example.

By forcing some of the connections and nodes in the learning networks620-623 to zero, the network 620-623 builds “collaterals” to compensate.In the process, insight into the topology of the learning network620-623 is obtained. Note that network 621 and network 622, for example,have different topologies since some nodes and/or connections have beenforced to zero. This process of effectively removing connections andnodes from the network extends beyond “deep learning” and can bereferred to as “deep-deep learning”, for example.

In certain examples, input data processing and deep learning stages canbe implemented as separate systems. However, as separate systems,neither module may be aware of a larger input feature evaluation loop toselect input parameters of interest/importance. Since input dataprocessing selection matters to produce high-quality outputs, feedbackfrom deep learning systems can be used to perform input parameterselection optimization or improvement via a model. Rather than scanningover an entire set of input parameters to create raw data (e.g., whichis brute force and can be expensive), a variation of active learning canbe implemented. Using this variation of active learning, a startingparameter space can be determined to produce desired or “best” resultsin a model. Parameter values can then be randomly decreased to generateraw inputs that decrease the quality of results while still maintainingan acceptable range or threshold of quality and reducing runtime byprocessing inputs that have little effect on the model's quality.

FIG. 7 illustrates example training and deployment phases of a learningnetwork, such as a deep learning or other machine learning network. Asshown in the example of FIG. 7 , in the training phase, a set of inputs702 is provided to a network 704 for processing. In this example, theset of inputs 702 can include features of an image to be identified. Thenetwork 704 processes the input 702 in a forward direction 706 toassociate data elements and identify patterns. The network 704determines that the input 702 represents a lung nodule 708. In training,the network result 708 is compared 710 to a known outcome 712. In thisexample, the known outcome 712 is a frontal chest (e.g., the input dataset 702 represents a frontal chest identification, not a lung nodule).Since the determination 708 of the network 704 does not match 710 theknown outcome 712, an error 714 is generated. The error 714 triggers ananalysis of the known outcome 712 and associated data 702 in reversealong a backward pass 716 through the network 704. Thus, the trainingnetwork 704 learns from forward 706 and backward 716 passes with data702, 712 through the network 704.

Once the comparison of network output 708 to known output 712 matches710 according to a certain criterion or threshold (e.g., matches ntimes, matches greater than x percent, etc.), the training network 704can be used to generate a network for deployment with an externalsystem. Once deployed, a single input 720 is provided to a deployedlearning network 722 to generate an output 724. In this case, based onthe training network 704, the deployed network 722 determines that theinput 720 is an image of a frontal chest 724. This same approach may beutilized in determining a tube or line, a reference or anatomicallandmark, and/or the proper placement of the tube or line

FIG. 8 illustrates an example product leveraging a trained networkpackage to provide a deep and/or other machine learning productoffering. As shown in the example of FIG. 8 , an input 810 (e.g., rawdata) is provided for preprocessing 820. For example, the raw input data810 is preprocessed 820 to check format, completeness, etc. Once thedata 810 has been preprocessed 820, it is fed into a trained network 840for processing. Based on learned patterns, nodes, and connections, thetrained network 840 determines outputs based on the input patches. Theoutputs are assembled 850 (e.g., combined and/or otherwise groupedtogether to generate a usable output, etc.). The output is thendisplayed 860 and/or otherwise output to a user (e.g., a human user, aclinical system, an imaging modality, a data storage (e.g., cloudstorage, local storage, edge device, etc.), etc.).

As discussed above, learning networks can be packaged as devices fortraining, deployment, and application to a variety of systems. FIGS.9A-9C illustrate various learning device configurations. For example,FIG. 9A shows a general learning device 900. The example device 900includes an input definition 910, a learning network model 920, and anoutput definition 930. The input definition 910 can include one or moreinputs translating into one or more outputs 930 via the network 920.

FIG. 9B shows an example training device 901. That is, the trainingdevice 901 is an example of the device 900 configured as a traininglearning network device. In the example of FIG. 9B, a plurality oftraining inputs 911 are provided to a network 921 to develop connectionsin the network 921 and provide an output to be evaluated by an outputevaluator 931. Feedback is then provided by the output evaluator 931into the network 921 to further develop (e.g., train) the network 921.Additional input 911 can be provided to the network 921 until the outputevaluator 931 determines that the network 921 is trained (e.g., theoutput has satisfied a known correlation of input to output according toa certain threshold, margin of error, etc.).

FIG. 9C depicts an example deployed device 903. Once the training device901 has learned to a requisite level, the training device 901 can bedeployed for use. While the training device 901 processes multipleinputs to learn, the deployed device 903 processes a single input todetermine an output, for example. As shown in the example of FIG. 9C,the deployed device 903 includes an input definition 913, a trainednetwork 923, and an output definition 933. The trained network 923 canbe generated from the network 921 once the network 921 has beensufficiently trained, for example. The deployed device 903 receives asystem input 913 and processes the input 913 via the network 923 togenerate an output 933, which can then be used by a system with whichthe deployed device 903 has been associated, for example.

In certain examples, condition identification (e.g., placement of a tubeor line) and progression can be determined through AI-driven analysis ofassociated image data for a patient.

FIG. 10 illustrates an example implementation of the AI classifier 220to process image data to be used by an AI model to quantify a condition(e.g., placement of a tube or line). The example implementation of theclassifier 220 enables annotation of one or more images including anorgan region and a region of interest within the organ region. Theexample classifier 220 of FIG. 10 includes an image segmenter 1010, amask combiner 1020, and a condition comparator 1040.

The example image segmenter 1010 is to identify a first mask and asecond mask in an input image. For example, the image segmenter 1010processes the image to segment a region of interest within an organregion identified in the image to obtain a first mask. The first mask isa segmentation mask and may be embodied as a filter that includes theregion of interest in the image and excludes the remainder of the image.The mask can be applied to image data to exclude all but the region ofinterest, for example. The mask can be obtained using a convolutionalneural network model, for example, such as the network 400, 500 shown inFIGS. 4-5 , a generative adversarial network, etc. The image segmenter1010 further processes the image to segment the organ region accordingto one or more criterion to obtain a second mask. For example, thesecond mask can represent the organ region, an area of the organ regionoutside the region of interest, etc.

For example, if the organ region is a lung (and the surrounding areasuch as the trachea), and the region of interest is a tube or lineidentified in the trachea, the first mask is generated to identify themedically placed tube or line, and the second mask is generated toidentify the entire organ region. In another embodiment, if the organregion is a stomach, and the region of interest is a tube or lineidentified in the in the stomach, the first mask is generated toidentify the medically placed tube or line, and the second mask isgenerated to identify the entire organ region. In a further embodiment,if the organ region is a heart (and the surrounding area such as veinsor other vasculature), and the region of interest is a tube or lineidentified in a vein or other vasculature near the heart, the first maskis generated to identify the medically placed tube or line, and thesecond mask is generated to identify the entire organ region. Thus, inregards to a medically placed tube or line, a first mask is generatedfor the tube or line and a second mask is generated for the entire organregion where the tube or line is placed (e.g., vasculature system,heart, lung, stomach, trachea, chest, pleural space, etc.).

The example combiner 1020 combines the first mask and the second maskand associated areas with annotation terms in the image. Annotations canbe relative qualification terms to produce quantification. For example,mask areas can be combined with descriptive terms such as foggy, patchy,dense, etc., to compute relative density values for the region ofinterest and organ region in the image. Image areas (e.g., areas offrontal and lateral images, etc.) can be combined to produce a volumemetric, for example.

The example distance computer 1030 determines a distance between asurface and/or an end of an identified tube or line and a reference oranatomical landmark (or determines a position of the tube or linerelative to the landmark). The example condition comparator 1040compares the distance or measured positions to a preset distance ordesired position for the type of tube or line and/or region of interestwhere the tube or line is placed (e.g., in accordance with predeterminedrules). Based on this comparison, the condition comparator 1040 candetermine whether the end of the tube or line is properly placedrelative to the reference or anatomical landmark.

Thus, the AI classifier 220 can be configured to annotate a medicalimage or set of related medical image(s) for AI/machine learning/deeplearning/CAD algorithm training, to quantify conditions. Such methodsare consistent, repeatable methodologies which could replace commonsubjective methods of today, enabling automatic, accurate detection ofthe presence of a medically placed tube or line and its placement.

While example implementations are illustrated in conjunction with FIGS.1-10 , elements, processes and/or devices illustrated in conjunctionwith FIGS. 1-10 can be combined, divided, re-arranged, omitted,eliminated and/or implemented in any other way. Further, componentsdisclosed and described herein can be implemented by hardware, machinereadable instructions, software, firmware and/or any combination ofhardware, machine readable instructions, software and/or firmware. Thus,for example, components disclosed and described herein can beimplemented by analog and/or digital circuit(s), logic circuit(s),programmable processor(s), application specific integrated circuit(s)(ASIC(s)), programmable logic device(s) (PLD(s)) and/or fieldprogrammable logic device(s) (FPLD(s)). When reading any of theapparatus or system claims of this patent to cover a purely softwareand/or firmware implementation, at least one of the components is/arehereby expressly defined to include a tangible computer readable storagedevice or storage disk such as a memory, a digital versatile disk (DVD),a compact disk (CD), a Blu-ray disk, etc. storing the software and/orfirmware.

A flowchart representative of example machine readable instructions forimplementing aspects or embodiments of the presently disclosedtechniques described herein are shown in conjunction with at least FIG.11 . In the examples, the machine readable instructions include aprogram for execution by a processor such as the processor 1312 shown inthe example processor platform 1300 discussed below in connection withFIG. 35 . The program may be embodied in machine readable instructionsstored on a tangible computer readable storage medium such as a CD-ROM,a floppy disk, a hard drive, a digital versatile disk (DVD), a Blu-raydisk, or a memory associated with the processor 1312, but the entireprogram and/or parts thereof could alternatively be executed by a deviceother than the processor 1312 and/or embodied in firmware or dedicatedhardware. Further, although the example program is described withreference to the flowchart illustrated in conjunction with at least FIG.11 , many other methods of implementing the components disclosed anddescribed herein may alternatively be used. For example, the order ofexecution of the blocks may be changed, and/or some of the blocksdescribed may be changed, eliminated, or combined. Although theflowchart of at least FIG. 11 depicts an example operation in anillustrated order, these operations are not exhaustive and are notlimited to the illustrated order. In addition, various changes andmodifications may be made by one skilled in the art within the spiritand scope of the disclosure. For example, blocks illustrated in theflowchart may be performed in an alternative order or may be performedin parallel.

As mentioned above, the example processes of at least FIG. 11 may beimplemented using coded instructions (e.g., computer and/or machinereadable instructions) stored on a tangible computer readable storagemedium such as a hard disk drive, a flash memory, a read-only memory(ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, arandom-access memory (RAM) and/or any other storage device or storagedisk in which information is stored for any duration (e.g., for extendedtime periods, permanently, for brief instances, for temporarilybuffering, and/or for caching of the information). As used herein, theterm tangible computer readable storage medium is expressly defined toinclude any type of computer readable storage device and/or storage diskand to exclude propagating signals and to exclude transmission media. Asused herein, “tangible computer readable storage medium” and “tangiblemachine readable storage medium” are used interchangeably. Additionallyor alternatively, the example processes of at least FIG. 11 can beimplemented using coded instructions (e.g., computer and/or machinereadable instructions) stored on a non-transitory computer and/ormachine readable medium such as a hard disk drive, a flash memory, aread-only memory, a compact disk, a digital versatile disk, a cache, arandom-access memory and/or any other storage device or storage disk inwhich information is stored for any duration (e.g., for extended timeperiods, permanently, for brief instances, for temporarily buffering,and/or for caching of the information). As used herein, the termnon-transitory computer readable medium is expressly defined to includeany type of computer readable storage device and/or storage disk and toexclude propagating signals and to exclude transmission media. As usedherein, when the phrase “at least” is used as the transition term in apreamble of a claim, it is open-ended in the same manner as the term“comprising” is open ended. In addition, the term “including” isopen-ended in the same manner as the term “comprising” is open-ended.

As mentioned above, these techniques may be utilized to identify amedically placed tube or line and to determine if medically placed tubeor line is properly placed. For example, the medically placed tube orline may be an enteric tube and the proper placement of the enteric tubemay be determined. Such examples are intended to be non-limiting, andany other tube or line inserted within a region of interest of the bodymay be identified and its proper placement determined.

FIG. 11 is a flow diagram of an embodiment of a method 1600 fordetermining a placement of a medically placed tube or line within aregion of interest. One or more steps of the method may be performed bythe processor platform 1600 in FIG. 35 . One or more steps of thedepicted steps may be performed simultaneously or in a different orderfrom what is illustrated in FIG. 11 . The method 1600 includes receivingor obtaining an image (e.g., chest image) of a patient that includes aregion of interest (ROI) (block 1602). The image may include a medicallyplaced tube or line inserted within the region of interest. The imagemay be provided while the patient has the tube or line inserted. Themethod 1600 also includes receiving or obtaining an input regarding thetype of tube or line to be detected (e.g., a nasogastric or otherenteric tube) and/or the region of interest for tube or line to beinserted within (e.g., trachea, stomach, gastrointestinal tract) (block1603). The input may be a user defined distance or rules for definingthe proper placement of the end of the medically placed tube or linerelative to a reference or anatomical location. In certain embodiments,the input may simply be the type of tube or line and/or the desiredregion of interest for the tube or line to be properly placed within.Based on this input, certain defined distances or rules (e.g., left,right, above, and/or below a specific anatomical location) may beutilized that define a proper placement of the end of the specific tubeor line within a specific region of interest. The method 1600 alsoincludes detecting the tube or line within the image (block 1604)utilizing the techniques described herein. The method 1600 includesidentifying a surface and/or an end (e.g., distal end) of the tube orline within the region of interest in the image (block 1606). The method1600 also includes identifying a reference or anatomical landmark withinthe image (block 1608). The reference or anatomical landmark will varybased on the type of tube or line utilized and the region of interestthat the tube or line is disposed within. For example, for a nasogastrictube, the reference or anatomical landmark(s) may include, but are notlimited to a patient's airway and diaphragm and/or a location within thestomach below the gastroesophageal junction.

Upon identifying the surfaces and/or distal end of the tube or line andthe reference or anatomical landmark(s), the method 1600 may includemeasuring a distance (e.g., between the end of the tube or line and thereference or anatomical landmark) (block 1610) that may be reported to auser and/or used in automated placement evaluation. The method 1600includes generating a combined image with indications of the tube orline, the reference or anatomical landmark(s), and/or the measureddistance (block 1612). Generating the combined image includessuperimposing various markers on the received image of the patient. Forexample, a color coding (e.g., color coded graphical overlay) may besuperimposed on the detected tube or line. In certain embodiments, thepatient may include more than one tube or line and the tube or line ofinterest is color coded. A graphical marker may be superimposed on theimage to indicate the end of the line or tube. Another graphical markermay be superimposed on the image to indicate the reference or anatomicallandmark. The graphical markers may include the same shape or differentshapes. Non-limiting examples of the shapes may be an open circle orother elliptical shape, open rectilinear shape, open triangular shape,or another shape. The graphical marker(s) and/or the tube may be colorcoded with different colors. For example, the graphical marker for thetube or line, the graphical marker for the reference or anatomicallandmark, and the tube or line may be green, blue, and yellow,respectively. A graphical marker may also be superimposed on the imageindicating a distance between the end of the tube or line and thereference or anatomical landmark when a distance is calculated. Thegraphical marker for the distance may also include the measurementvalue. The method 1600 further includes displaying the combined image ona display (block 1614). The combined image may be displayed in real-timeto the medical personnel enabling them to adjust the placement of thetube or line if need be. In certain embodiments, the combined image maybe displayed as a DICOM image.

In certain embodiments, the method 1600 includes calculating one or morerespective confidence metrics (block 1616). The confidence metrics maybe for the calculated distance, for the determination of the presence ofthe medically placed tube or line, for an accuracy in detecting theplacement of the tube or line, and/or for an accuracy in detecting thereference or anatomical landmark. The confidence metric may include aconfidence level or confidence interval. The confidence metric may bestored for future reference. In certain embodiments, the method 1600 mayinclude providing one or more of the confidence metrics to a user (block1618). For example, the confidence metrics may be displayed on thecombined image or provided on a separate device (e.g., user's device).In certain embodiments, the confidence metrics may be written into astandard or private information tag (e.g., DICOM) and made visible insubsequent information systems that the image is sent too (e.g., PACS).

In certain embodiments, in determining whether the medically placed tubeor line is placed properly (e.g., via the deep learning networksmodels), the method 1600 includes comparing the measured distancebetween the surface and/or end of the tube or line and the reference oranatomical landmark to a desired threshold (block 1620) and determiningif the distance is acceptable (block 1622). The desired threshold mayrepresent an acceptable range for the distance between the tube or lineand the reference or anatomical landmark for the tube or line to becorrectly placed. For example, for a nasogastric tube, the desiredthreshold may be a range of distance below the gastroesophagealjunction. If the measured distance is not acceptable, the method 1600includes providing a user-perceptible indication of misplacement (block1624). The indication may be provided on the display where the combinedimage is displayed or provided on another device (e.g., the user'sdevice). The indication may be text stating that the tube or line ismisplaced. In certain embodiments, the text may be more specific andstate the tube or line is too high or too low or otherwise improper. Incertain embodiments, the text may provide further instructions (e.g., toraise or lower the end of the tube or line a certain distance). In someembodiments, the text may be color coded (e.g., in orange or red) tofurther indicate the misplacement. In some embodiments, the indicationmay be provided via color coding of one or more graphical markers or thetube or line displayed on the combined image. For the example, one ormore of the graphical markers (e.g., for the end of tube or line, forthe reference or anatomical landmark, and/or the indication of themeasured distance there between) and/or the tube or line may be colorcoded a specific color (e.g., red or orange) to indicate themisplacement. Alternatively or in addition, one or more of the graphicalmarkers may flash or otherwise be visually highlighted if the tube orline is misplaced. If the measured distance is acceptable, the method1600 includes providing a user-perceptible indication of properplacement of the tube or line (block 1626). The indication may beprovided on the display where the combined image is displayed orprovided on another device (e.g., the user's device). The indication forproper placement may be text stating the tube or line is properlyplaced. In certain embodiments, the indication for proper placement maybe provided via color coding one or more graphical markers of the tubeor line displayed on the combined image (e.g., all the graphical markersand/or the tube or line may be color coded green). In certainembodiments, the indication of proper placement or misplacement may bewritten into a standard or private information tag (e.g., DICOM) andmade visible in subsequent information systems that the image is senttoo (e.g., PACS). In certain embodiments, the determination as towhether the medically placed tube or line is properly placed orpositioned may be manually done by the medical personnel viewing thedisplayed combined image.

With the preceding in mind, and by way of a real-world context andexample to facilitate explanation, further illustration of an enterictube implementation is described below. As used herein, enteric tubesmay be understood to be thin flexible hollow catheters that course intothe stomach and beyond. In practice, such enteric tubes may or may notinclude a side port. As may be appreciated, the phrase “enteric tube”may be understood to encompass an array of tube types differentiated bytheir insertion point (e.g., in the nose (naso-) or mouth (oro-) and bytheir endpoint (e.g., in the stomach (-gastric), in the duodenum(-duodenal), or in the jejunum (-jejunal). For the purpose ofillustration, many of the following examples are presented in thecontext of a nasogastric tube so as to provide a real-world context.However it should be understood that such examples and discussion may beequally applicable to the other types of enteric tubes and, indeed, toother suitable medical tubes in general.

In the context of a nasogastric tube, it may be understood that the useof such tubes may raise particular issues which may be addressed asexplained and shown herein. By way of context, such tubes may typicallybe implemented as plastic (or other biocompatible material) tubes thatare designed to be passed through the nose and into the stomach of apatient. Once properly placed, a nasogastric tube may be used toadminister nutrients medication, and/or contrast to the patient. Inaddition or in the alternative, the placed tube may be used to removeliquids and/or air from the stomach.

With respect to proper placement, a nasogastric tube may be inserted soas to bisect the airways and diaphragm on the X-ray projection (e.g., tobe positioned substantially on the midline with respect to the airway).The inserted tip (i.e., distal tip) and side ports (if present) arebelow the diaphragm when properly placed, typically positioned towardthe patient's left hand side. Proper insertion and placement of the tubeavoids or mitigates possible risks, such as the risk of insertion intothe lungs (with the associated risk of substances entering the lungs),the risk of the tube placement being too high, e.g., in the esophagus,and the risk that loops or kinks in the inserted tube may disturb theflow and/or irritate the patient.

As discussed herein, and in the context of the preceding discussion, thepresently described techniques utilize an AI-based feature to facilitateand assess the placement of enteric tubes, including but not limited tonasogastric tubes. The AI-based feature may be used to detect and/orcharacterize the placed tube, to provide a graphical summary showing thetube with respect to relevant anatomical features (e.g., in the actualanatomical context), and to classify the tube as being placed correctlyor needing adjustment. Use of the AI-based feature may, therefore,increase the confidence of the bedside team when placing tubes. Use ofthe Ai-based feature may also facilitate prioritization of potentiallymisplaced tubes for review, such as by a radiologist, and may speed upthe review process, thereby helping to avoid complications associatedwith misplaced tubes.

Features and benefits provided by the techniques described hereininclude, but are not limited to: the ability to localize particularfeatures (e.g., the tube tip, side port, end port, and so forth) of theenteric tube; the ability to localize relevant anatomical features andcontext (e.g., diaphragm, airways, carina, lungs, patient midline, andso forth); the ability to localize other relevant devices that may bepotentially confounding with enteric tubes (e.g., probes, peripherallyinserted central catheter (PICC) lines, electrocardiogram (ECG) leads orlines, endotracheal (ET) tube, and so forth); the ability to assess thetube position and to provide explanation or commentary about theassessment (e.g., explaining specific problems with current tubeplacement, such as “the side port location is too high relative to thediaphragm”); the ability to assess the tube position and to provideexplanation or commentary regarding aspects of the placement verified tobe correct or satisfactory (e.g., that the tube correctly bisects thediaphragm near the midline); the ability to provide automatedmeasurements that are relevant for the tube assessment (e.g. the lengthof the tube below the diaphragm, the distance of the side port from thediaphragm, the measured tube diameter, etc.); the ability to show thedetected tubes, the tube features, and relevant anatomical features andmeasurements in a graphical summary and the ability to highlightpotentially problematic (or non-problematic) areas within the graphicalsummary); the ability to perform triage based on the tube placementclassification, allowing prioritization of attention to potentiallymisplaced tubes; the ability to save the graphical summary in variousformats (secondary capture, structure report, Integrating the HealthcareEnterprise (IHE) AI Results (AIR), and so forth); and the ability toallow the user to edit, modify, and/or annotate the graphical summary.

It may be noted that the present techniques, as applied to enterictubes, may address additional complexity as compared to approaches thatrelate primarily to endotracheal tubes or other medical tubes. By way ofexample, the present techniques as applied to enteric tubes must addressor allow for the acquisition and use of both chest and abdominal images(as opposed to only chest images). Further, enteric tubes have a varietyof tube types which must be taken into account. By way of example,nasogastric tubes encompass standard nasogastric tubes (e.g., Levintubes, Ryle's tubes), Salem sump tubes, small-bore silicone rubberfeeding tubes (e.g., Keofeed tubes, Duo-tubes, Dobbhoff tubes), andother special purpose tubes (e.g., Ewald tubes, Cantor tubes,Miller-Abbott tubes, Sengstaken-Blakemore tubes, Minnesota tubes,Linton-Nachlas tubes, Nutrivent tubes). In addition, enteric tubes,unlike endotracheal tubes, may form loops and complex curves wheninserted and may allow for many potential trajectories. Correspondingly,enteric tubes may have many placement requirements (relative to othermedical tube insertions) and may be subject to many and varied types ofmisplacement. Due to their length, certain types of enteric tubes mayalso exit and/or re-enter the acquired images, posing a further distinctcomplexity to be addressed.

By way of context, a high level workflow for an enteric tube (here anasogastric tube (NGT)) placement is illustrated in FIG. 12 . In thisexample, at step 1700 a point (e.g., an anatomical point) is indicatedfor insertion of the nasogastric tube. The nasogastric tube may then bemarked for the appropriate insertion distance (step 1702). Thenasogastric tube may then be inserted or otherwise placed (step 1704) bymedical personnel, such as a gastrointestinal nurse or bedside doctor.Once inserted, the nasogastric tube may be secured (step 1706) to theface and/or neck of the patient. To verify placement in the stomach, afluid sample may be aspirated and tested for pH (step 1708). An X-rayimaging operation may be performed (step 1710) to assess positioning ofthe nasogastric tube. The X-ray image data may be assessed and/orpresented in accordance with the techniques discussed herein. Theoutputs of the analysis of the X-ray image data, as shown in FIG. 12 maybe provided to the bedsides clinician(s) (step 1712) to assess theposition of the nasogastric tube and to make any needed adjustments.

The outputs of the analysis of the X-ray image data, as shown in FIG. 12may, in addition or in the alterative, be provided (step 1714) to apictures archiving systems (PACS) for review by a radiologist. In thedepicted example, two outcomes are illustrated with respect toradiologist review. In the first (step 1718), the tube placement isdetermined to be optimal or satisfactory and a radiologist report isgenerated to this effect, which may measurement data. In the second(step 1716), the tube is determined to be misplaced and this informationis communicated to the bedside team. The tube may then be repositioned(step 1720) by the bedside team and additional X-rays acquired (step1722) to assess tube placement. Benefits of utilizing the techniques asdiscussed herein with respect to this workflow include, but are notlimited to: providing increased confidence in tube placement, providingalerts as to misplaced tubes and/or suboptimal image quality,prioritizing cases of suspected tube misplacement, providing automatedmeasurements that might not otherwise be generated, and speeding upreport creation. Thus, as described herein, the presently describedtechnique helps detect the presence and /or number of enteric tubes,helps determine if the enteric tube is malpositioned, provides animmediate bedside alert for malpositioned tubes, provides PACSprioritization for suspected malpositioned tubes, and providesvisualization of the tube and anatomical context and features. In termsof clinical benefits, these may include immediate bedside interventionin critical cases based on the decision of the bedside clinician, fasterintervention in critical cases based on the radiology report, and higherconfidence for the bedside team for appropriately placed tubes.

Turning to FIG. 13 , a brief overview of a model architecture forsegmenting and classifying an enteric tube (e.g., a nasogastric tube).In this example, an image 1800 or sequence of images of the anatomicregion of interest (e.g., chest and abdomen) is acquire and provided toa backbone network 1804. In the depicted example, the image(s) 1800include an enteric tube 1802 as part of the image. Image data andoperations are shown as being performed in the context of the network1804. A segmentation operation 1806 may be performed to identify andsegment the tube 1802 within each image 1800 so as to generate asegmented object or representation 1818 of the tube 1802.

Classification logic (such as AI-based classification operations) mayoperate on the segmented representation 1818 to generate a series ofclassification outputs. By way of example, classification logic 1800 maymake an initial determination as to whether a tube 1802 is present inthe image 1800 based on the provided inputs. In addition, assuming atube 1802 is determined to be present, further classification logic 1812and segmentation logic 1814 may be performed to determine if a tube tipis present and, correspondingly, to localize and segment the tube tip1820 within the image(s) 1800. Further classification logic 1808 maydetermine, based upon the segmented representation and in combinationwith a specified maximum value or threshold 1822, whether the tube 1802extends beyond the image 1800. Classification logic 1816 may also beapplied to detect whether loops are present in the tube 1802 based uponthe segmented representation 1818. In view of this architecture, eachimage or sequence of images 1800 may be processed as described herein toprovide information to the bedside clinicians and/or to radiologistsviewing the images and outputs via PACS.

As discussed herein, outputs of the AI-based logic may be used to assessor otherwise evaluate placement of an enteric tube. For example, outputsof the AI-based logic may be utilized to characterize a placement asexpected or satisfactory (e.g., side port okay, no side port long, tipoutside long), as having a loop or kink (e.g., loop then down, kink, toodeep), as being borderline (e.g., tip outside short, sideportborderline, tube length borderline), as being malpositioned so as topose an airways risk (e.g., above carina, in airways), as beingmalpositioned so as to be too high (e.g., in esophagus, no side portshort, side port high), as being malpositioned so as to be high withloops (e.g., loop above diaphragm, returns to esophagus), or as having alimited field-of-view or otherwise out-of-scope (e.g., tip outsideshort, below diaphragm short, below diaphragm exit up). In practice, andwith the preceding in mind, there may be a number of varied and suitableoptions for grouping the potential tube positions into possibleclassification outputs for a given implementation including, but notlimited to: (1) malpositioned tube present/no malpositioned tubepresent, (2) no tube present/correctly placed tube/malpositioned tube,(3) no tube present/correctly placed tube/tube position needsverification, (4) no tube present/correctly placed tube/malpositionedtube/borderline placement/partially visualized tube/loops or kinkspresent/out of scope due to limited field of view, (5) no tubepresent/correctly placed tube/correctly placed tube with sideport/malpositioned tube in airways/malpositioned tube inesophagus/malpositioned tube too high/borderline placement/loops orkinks present/partially visualized tube/uncertain/out of scope due tolimited field of view/and so forth.

With regard to the presented or displayed information, and turning toFIGS. 14 and 15 , an example of a combined image 1628 (e.g., DICOMimage) identifying an enteric tube or line within a patient that may bedisplayed on a display. As depicted, the combined image 1628 of apatient shows a nasogastric tube 1632 positioned within the patient. Aspart of the combined image, one or more anatomical references orfeatures may be indicated via overlay of dashed lines or other visualreferences to facilitate user visualization of the tube 1632 relative tothe anatomic context. By way of example, in the depicted image dashedline 1660 is overlaid to illustrate the bronchi, dashed line 1662 isoverlaid to illustrate the patient midline, and dashed line 1664 isoverlaid to illustrate the diaphragm.

In the depicted example, a graphical marker 1634 (e.g., circle orcolor-coded circle) overlaid on the combined image 1628 indicates thelocation of the end (i.e., tube tip) of the nasogastric tube 1632. Agraphical marker 1636 (e.g., circle or color-coded circle) overlaid onthe chest image indicates a reference or anatomical location (e.g.,carina). A graphical marker 1630 (e.g., circle, color-coded circle,dashed circle, and so forth) overlaid on the image indicates a side holeor side port, if present, of the nasogastric tube 1632. A numericalvalue 1640 indicates a measured distance (e.g., a tube length distance)between the tip of the nasogastric tube 1632 and a reference oranatomical location, here the diaphragm. In certain embodiments, aconfidence metric in the measured distance generated by the artificialintelligence is also displayed (e.g., as depicted a confidence level).In certain embodiments, the tube 1632, the graphical markers 1630, 1634,and/or 1636 may be color coded (e.g., blue, yellow, green, and red) orotherwise visually coded (e.g., solid line, dashed line, double lines,lines of distinctive thickness).

FIG. 15 is a schematic diagram of a user interface 1652 having acombined image 1628 identifying a tube or line within a patient that maybe displayed on a display. As shown in FIG. 15 , the combined image 1628may also include or be displayed alongside a header or other informationblock or section 1642 that includes information related to the image1628. For example, as depicted, the information block 1642 (a portion ofwhich is expanded as an inset to improve visibility) includes the typeof tube or line 1644 (e.g., a nasogastric tube), whether the placementof the tube is proper or not 1646, and the calculated distance 1648between the tip of the tube and the reference or anatomical marker.

In certain embodiments, the information block 1642 may include anindication as to whether the tube or line was detected, such as by theAI-based logic discussed herein. In certain embodiments, one or moreconfidence metrics may be displayed on the image 1628 (e.g., for thecalculated distance, for the determination of the presence of themedically placed tube or line, for an accuracy in detecting the tip ofthe tube or line, and/or for an accuracy in detecting the reference oranatomical landmark). As shown in the example, in certain embodimentsthe information block may also include other relevant placementinformation determined by the AI-based logic. By way of example, in thedepicted information block of FIG. 15 information about whether the tube1632 bisect the bronchi, whether the tube crosses the diaphragm nearmidline, whether the tip is below the diaphragm on the left, whether theside hole is below the diaphragm, the tip distance from the diaphragm,and whether the tube curvature appears normal are all displayed. Thatis, in certain embodiments, the depicted indication may indicate whetherthe tube or line is placed properly, misplaced, provide an indication ofwhat is wrong with the placement, or provide instructions for correctingthe placement. An option to turn the image overlay on or off is alsoprovided in the depicted example.

By way of providing further examples, additional user interfaces 1652are described below that illustrate different tube placements and AIoutcomes and how such placements and outcomes might be presented to auser. By way of example, and turning to FIG. 16 , an example isdisplayed of results and visualization provided for a normal or expectedtube placement with a side port present. FIG. 17 depicts an example ofresults and visualization for a normal or expected tube placement whereno side port is present and with a long placement with respect to thetip extending beyond the diaphragm. FIG. 18 depicts an example ofresults and visualization for a normal or expected tube placement whereno side port is present and with a long placement where the tip is atthe edge of or outside the image. FIG. 19 depicts an example of resultsand visualization for a normal or expected tube placement where the tube1632 is in a long placement below the diaphragm. FIG. 20 depicts anexample of results and visualization for a normal or expected tubeplacement where the tube 1632 is placed deep but only a short portion ofthe tube is visible.

While the preceding examples illustrate expected or acceptableplacements, the following examples illustrate malpositioned, borderline,or indeterminate tube placements. Turning to FIG. 21 , an example isdisplayed of results and visualization provided for a malpositioned tubeplacement in which the side port is high (e.g., positioned above thediaphragm). To facilitate communication of the problematic aspect of thetube placement, one or both of the graphical marker 1630 correspondingto the side port and/or the graphical marker (e.g., dashed line) 1664corresponding to the diaphragm may be color-coded (e.g., yellow or red).Similarly the text 1646 in the information block 1642 indicating whetherthe placement of the tube is proper or not may indicate that the tube ismalpositioned. This text may also be color-coded (e.g., yellow or red)to indicate a problematic placement. Similarly, FIG. 22 depicts anexample of results and visualization for a malpositioned tube placementwhere no side port is present and with a short placement with respect tothe tip extending beyond the diaphragm. FIG. 23 depicts an example ofresults and visualization for a malpositioned tube placement where thetube tip is positioned above the carina. FIG. 24 depicts an example ofresults and visualization for a malpositioned tube placement where thetube 1632 is positioned within the esophagus. FIG. 25 depicts an exampleof results and visualization for a malpositioned tube placement wherethe tube 1632 is positioned within airways. FIG. 26 depicts an exampleof results and visualization for a malpositioned tube placement wherethe tube 1632 is positioned within the pleural cavity. FIG. 27 depictsan example of results and visualization for a malpositioned tubeplacement where the tube 1632 is positioned so as to return andre-ascend the esophagus. FIG. 28 depicts an example of results andvisualization for a malpositioned tube placement where the tube 1632 ispositioned so as to loop above the diaphragm.

In the following examples, a suboptimal tube placement is illustrated inthe context of the AI outputs and provided visualizations. Turning toFIG. 29 , an example is displayed of results and visualization providedfor one such suboptimal position in which a tube segment above thediaphragm loops, but the side port and tip placement is below thediaphragm. Similarly, FIG. 30 depicts an example of results andvisualization for a suboptimal tube placement where a tube segment loopsbelow the diaphragm and the overall tube placement is too deep. FIG. 31depicts an example of results and visualization for a suboptimal tubeplacement where a tube segment loops is kinked below the diaphragm. Tofacilitate communication of the problematic aspect of the tubeplacement, the problematic segment 1666 of the tube 1632 may becolor-coded (e.g., yellow or red).

In the following examples, the tube is only partially visualized in theimage data. Turning to FIG. 32 , an example is displayed of results andvisualization provided for one such partially visualized tube in whichthe tube tip is below the diaphragm but for a limited or short length.FIG. 33 depicts an example of results and visualization for a partiallyvisualized tube where tube ascends back up from below the diaphragm toterminate out of the image. FIG. 34 depicts an example of results andvisualization for a partially visualized tube where tube extends beyondthe image frame.

FIG. 35 is a block diagram of an example processor platform 1300structured to executing the processor-executable instructions toimplement the example components disclosed and described herein. Theprocessor platform 1300 can be, for example, a server, a personalcomputer, a mobile device (e.g., a cell phone, a smart phone, a tabletsuch as an iPad™), a personal digital assistant (PDA), an Internetappliance, or any other type of computing device.

The processor platform 1300 of the illustrated example includes aprocessor 1312. The processor 1312 of the illustrated example ishardware. For example, the processor 1312 can be implemented byintegrated circuits, logic circuits, microprocessors or controllers fromany desired family or manufacturer.

The processor 1312 of the illustrated example includes a local memory1313 (e.g., a cache). The example processor 1312 of FIG. 35 executesprocessor-executable instructions to implement the systems,infrastructure, displays, and associated methods of FIGS. 1-13 such asthe example data source 210, AI classifier 220, data store 230,comparator 240, output generator 250, trigger 260, etc. The processor1312 of the illustrated example is in communication with a main memoryincluding a volatile memory 1314 and a non-volatile memory 1316 via abus 1318. The volatile memory 1314 may be implemented by SynchronousDynamic Random Access Memory (SDRAM), Dynamic Random Access Memory(DRAM), RAMBUS Dynamic Random Access Memory (RDRAM) and/or any othertype of random access memory device. The non-volatile memory 1316 may beimplemented by flash memory and/or any other desired type of memorydevice. Access to the main memory 1314, 1316 is controlled by a clockcontroller.

The processor platform 1300 of the illustrated example also includes aninterface circuit 1320. The interface circuit 1320 may be implemented byany type of interface standard, such as an Ethernet interface, auniversal serial bus (USB), and/or a PCI express interface.

In the illustrated example, one or more input devices 1322 are connectedto the interface circuit 1320. The input device(s) 1322 permit(s) a userto enter data and commands into the processor 1312. The input device(s)can be implemented by, for example, a sensor, a microphone, a camera(still or video, RGB or depth, etc.), a keyboard, a button, a mouse, atouchscreen, a track-pad, a trackball, isopoint and/or a voicerecognition system.

One or more output devices 1324 are also connected to the interfacecircuit 1320 of the illustrated example. The output devices 1324 can beimplemented, for example, by display devices (e.g., a light emittingdiode (LED), an organic light emitting diode (OLED), a liquid crystaldisplay, a cathode ray tube display (CRT), a touchscreen, a tactileoutput device, and/or speakers). The interface circuit 1320 of theillustrated example, thus, typically includes a graphics driver card, agraphics driver chip or a graphics driver processor.

The interface circuit 1320 of the illustrated example also includes acommunication device such as a transmitter, a receiver, a transceiver, amodem and/or network interface card to facilitate exchange of data withexternal machines (e.g., computing devices of any kind) via a network1326 (e.g., an Ethernet connection, a digital subscriber line (DSL), atelephone line, coaxial cable, a cellular telephone system, etc.).

The processor platform 1300 of the illustrated example also includes oneor more mass storage devices 1328 for storing software and/or data.Examples of such mass storage devices 1328 include floppy disk drives,hard drive disks, compact disk drives, Blu-ray disk drives, RAIDsystems, and digital versatile disk (DVD) drives.

Coded instructions may be stored in the mass storage device 1328, in thevolatile memory 1314, in the non-volatile memory 1316, and/or on aremovable tangible computer readable storage medium such as a CD or DVD.

From the foregoing, it will be appreciated that the above disclosedmethods, apparatus, and articles of manufacture have been disclosed tomonitor, process, and improve operation of imaging and/or otherhealthcare systems using a plurality of deep learning and/or othermachine learning techniques.

Thus, certain examples facilitate image acquisition and analysis at thepoint of care such as via an imaging device at the point of patientimaging. If images should be re-taken, further analysis done right away,and/or other criticality explored sooner, rather than later, the examplesystems, apparatus, and methods disclosed and described herein canfacilitate such action to automate analysis, streamline workflow, andimprove patient care.

Certain examples provide a specially-configured imaging apparatus thatcan acquire images and operate as a decision support tool at the pointof care for a critical care team. Certain examples provide an imagingapparatus that functions as a medical device to provide and/orfacilitate diagnosis at the point of care to detect radiologicalfindings, etc. The apparatus can trigger a critical alert for aradiologist and/or critical care team to bring immediate attention tothe patient.

In certain examples, a mobile device and/or cloud product enables avendor-neutral solution, proving point of care alerts on any digitalx-ray system (e.g., fully integrated, upgrade kit, etc.). In certainexamples, embedded AI algorithms executing on a mobile imaging system,such as a mobile x-ray machine, etc., provide point of care alertsduring and/or in real-time following image acquisition, etc.

By hosting AI on the imaging device, a mobile x-ray system can be usedin rural regions without hospital information technology networks, oreven on a mobile truck that brings imaging to patient communities, forexample. Additionally, if there is long latency to send an image to aserver or cloud, AI on the imaging device can instead be executed andgenerate output back to the imaging device for further action. Ratherthan having the x-ray technologist moved onto the next patient and thex-ray device no longer at the patient's bedside with the clinical careteam, image processing, analysis, and output can occur in real time (orsubstantially real time given some data transfer/retrieval, processing,and output latency) to provide a relevant notification to the clinicalcare team while they and the equipment are still with or near thepatient. For trauma cases, for example, treatment decisions need to bemade fast, and certain examples alleviate the delay found with otherclinical decision support tools.

Mobile X-ray systems travel throughout the hospital to the patientbedside (e.g., emergency room, operating room, intensive care unit, etc.Within a hospital, network communication may be unreliable in “dead”zones of the hospital (e.g., basement, rooms with electrical signalinterference or blockage, etc.). If the X-ray device relies on buildingWi-Fi, for example, to push the image to a server or cloud which ishosting the AI model and then wait to receive the AI output back to theX-ray device, then patient is at risk of not having reliability incritical alerts when needed. Further, if a network or power outageimpacts communications, the AI operating on the imaging device cancontinue to function as a self-contained, mobile processing unit.

Examples of alerts generated for general radiology can include criticalalerts (e.g., for mobile x-ray, etc.) such as tubes and line placement,pleural effusion, lobar collapse, pneumoperitoneum, pneumonia, etc.;screening alerts (e.g., for fixed x-ray, etc.) such as tuberculosis,lung nodules, etc.; quality alerts (e.g., for mobile and/or fixed x-ray,etc.) such as patient positioning, clipped anatomy, inadequatetechnique, image artifacts, etc.

Thus, certain examples improve accuracy of an artificial intelligencealgorithm. Certain examples factor in patient medical information aswell as image data to more accurately predict presence of a criticalfinding, an urgent finding, and/or other issue.

Although certain example methods, apparatus and articles of manufacturehave been described herein, the scope of coverage of this patent is notlimited thereto. On the contrary, this patent covers all methods,apparatus and articles of manufacture fairly falling within the scope ofthe claims of this patent.

Technical effects of the disclosed subject matter include providingsystems and methods that utilize AI (e.g., deep learning networks) todetermine whether or not a medically placed tube or line is properlyplaced within a region of interest (e.g., relative to a reference oranatomical landmark). The systems and methods may provide feedback inreal time that in a more accurate and quicker manner determine if amedically placed tube or line is misplaced. Thus, enabling fastintervention, if needed, to move the tube or line to the appropriatelocation for patient safety.

This written description uses examples to disclose the subject matter,including the best mode, and also to enable any person skilled in theart to practice the subject matter, including making and using anydevices or systems and performing any incorporated methods. Thepatentable scope of the disclosed subject matter is defined by theclaims, and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims if they have structural elements that do not differ from theliteral language of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

1. A medical image processing system, comprising: a display; aprocessor; and a memory storing processor-executable code that whenexecuted by the processor causes: receiving one or both of a chest orabdominal image of a patient with an enteric tube or line disposedwithin the region of interest; detecting the enteric tube or line withinthe image or images; detecting one or more anatomic reference landmarkswithin the region of interest; generating a combined image bysuperimposing one or more graphical markers on the image or imageswherein the one or more graphical markers indicate one or more featuresof the enteric tube or line in an anatomic context derived based on theone or more anatomic reference landmarks; and displaying the combinedimage on the display.
 2. The medical image processing system of claim 1,wherein generating the combined image comprises superimposing one ormore anatomic graphical markers on the image or images that indicatesthe anatomic reference landmarks.
 3. The medical image processing systemof claim 2, wherein the processor-executable code when executed by theprocessor causes calculating a measured position of a distal tip of theenteric tube or line relative to a respective anatomic referencelandmark, and wherein generating the combined image comprisessuperimposing a graphical indicator on the image or images thatindicates the measured position.
 4. The medical image processing systemof claim 1, wherein the processor-executable code when executed by theprocessor causes determining whether the enteric tube or line is whollyor partially visible within the image.
 5. The medical image processingsystem of claim 1, wherein the processor-executable code when executedby the processor causes determining whether the enteric tube or line isproperly placed relative to a respective anatomic reference landmark. 6.The medical image processing system of claim 5, wherein theprocessor-executable code when executed by the processor causesdisplaying an indication with the combined image when the enteric tubeor line is misplaced relative to the respective anatomic referencelandmark.
 7. The medical image processing system of claim 1, wherein theenteric tube or line comprises a nasogastric tube, a nasoduodenal tube,a nasojejunal, tube, an orogastric tube, an oroduodenal tube, or anorojejunal tube.
 8. The medical image processing system of claim 1,wherein the one or more anatomic reference landmarks comprises a carina,a diaphragm, an airway, a stomach, lungs, a midline, or an edge of anairway.
 9. The medical image processing system of claim 1, whereindetecting the enteric tube or line within the image or images comprisesutilizing one or more deep learning network models to detect the enterictube or line within the image or images.
 10. The medical imageprocessing system of claim 1, wherein generating the combined imagecomprises superimposing on the image a colored graphical overlay overthe enteric tube or line to facilitate visualization of the enteric tubeor line.
 11. The medical image processing system of claim 1, wherein theone or more graphical markers are color-coded to indicate one or both ofproper or improper placement of the enteric tube or line.
 12. A methodfor medical image processing, comprising: receiving, via a processor,one or both of a chest or abdominal image of a patient with an enterictube or line disposed within the region of interest; detecting, via theprocessor, the enteric tube or line within the image or images;detecting one or more anatomic reference landmarks within the region ofinterest; generating, via the processor, a combined image bysuperimposing one or more graphical markers on the image or imageswherein the one or more graphical markers indicate one or more featuresof the enteric tube or line in an anatomic context derived based on theone or more anatomic reference landmarks; and causing, via theprocessor, display of the combined image on a display.
 13. The method ofclaim 12, comprising detecting, via the processor, of one or moreanatomic reference landmarks within the region of interest.
 14. Themethod of claim 13, wherein generating the combined image comprisessuperimposing, via the processor, one or more anatomic graphical markerson the image or images that indicates the anatomic reference landmark.15. The method of claim 14, comprises calculating, via the processor, ameasured position of a distal tip of the enteric tube or line relativeto a respective anatomic reference landmark, and wherein generating thecombined image comprises superimposing a graphical indicator on theimage or images that indicates the measured position.
 16. The method ofclaim 13, comprises determining, via the processor, whether the enterictube or line is wholly or partially visible within the image.
 17. Themethod of claim 13, comprising determining, via the processor, whetherthe enteric tube or line is properly placed relative to a respectiveanatomic reference landmark.
 18. The method of claim 17, comprisingdisplaying, via the processor, an indication with the combined imagewhen the enteric tube or line is misplaced relative to the respectiveanatomic reference landmark.
 19. The method of claim 12, whereindetecting the enteric tube or line within the image or images comprisesutilizing, via the processor, one or more deep learning network modelsto detect the enteric tube or line within the image or images.
 20. Anon-transitory, computer-readable medium, the computer-readable mediumcomprising processor-executable code configured to: receive one or bothof a chest or abdominal image of a patient with an enteric tube or linedisposed within the region of interest; detect the enteric tube or linewithin the image or images; detect one or more anatomic referencelandmarks within the region of interest; generate a combined image bysuperimposing one or more graphical markers on the image or imageswherein the one or more graphical markers indicate one or more featuresof the enteric tube or line in an anatomic context derived based on theone or more anatomic reference landmarks; and display the combined imageon a display.